Special Topics Case Study

Research & Activity Documentation

Khant Nyar Lu

---

Introduction

---

I've been producing music for almost four years, so when this course gave me room to explore the intersection of music and interaction design, it felt like the right fit. What I didn't expect was how naturally TouchDesigner would connect to what I already know from Ableton. The node-based workflow, the signal routing, the idea of patching things together: it clicked faster than I thought it would.

Working with Bre made the whole semester easier to navigate. She came in stronger on the visual side; I had more to offer on the audio and technical side. That split turned out to be genuinely useful. We spent a lot of Module 2 asking each other questions across our areas, and by the time we got to the final project, building together made more sense than doing separate things. So instead of one big joint project, we built two smaller collaborative systems where we each led on different parts. One system was more visual, one was more audio. Both of us touched both.

Module 1: Concept

---

Research

Module 1 was mostly about figuring out what we wanted to do. We looked at how TouchDesigner and Ableton get used in live performance contexts: concerts, installations, AV sets. We tried to understand what techniques were actually worth learning. The goal wasn't to find one perfect reference; it was to get a feel for the space so we could start making decisions about direction.

Our design statement came out of that research: explore gesture-based musical interaction through technical work in TouchDesigner, with a focus on technique over a polished final installation. We wanted to build something real and functional rather than demo-y. The idea of learning skills that could eventually transfer to stage AV or interactive exhibits was part of the thinking from pretty early on.

Project 1: Concept

---

The concept was straightforward: explore different types of interaction we could layer into a music experience, focusing on actually learning the technique rather than shipping a polished installation. We were both new to parts of this toolchain, so getting things working and understanding why they worked felt more important than aesthetics at this stage.

The bigger picture was always live AV performance: concerts, festivals, raves, museum installs. We kept that in the back of our heads as a way to check whether the skills we were building were worth building.

Module 2: Prototype

---

Research

Module 2 is where I actually started learning TouchDesigner. My approach was basically: find tutorials, replicate them, figure out what I don't understand in the process. It's slower than just watching, but it stuck a lot better. By the end of the module I had three working systems; one music visualizer, one hand tracking demo, one Ableton bridge. That was enough of a feel for the tool to start doing things off-script.

Bre was working through a similar process from the other direction. She focused on getting Ableton basics down and learning MediaPipe hand tracking through the TouchDesigner plugin. By Module 2's end she had two separate hand-tracking systems running with audio and visual components. We compared notes a lot during this phase, which helped both of us move faster than we probably would have working in isolation.

Project 2: Prototype

---

The prototype phase ended up being two things: a music-reactive visualizer and a gesture-controlled Ableton rig. Neither was finished enough to present as a final project, but both taught me enough about the tools that the final project felt achievable. That was the point.

It also made it clear where Bre and I were strongest. She had a better eye for how the visuals should look and feel. I had more intuition for the audio routing and parameter mapping. That defined how we split the work in Project 3.

Module 3: Product

---

Research

For Module 3, two specific references shaped what we built. For System A, we were looking at @musicazura and @elinvarstudio on Instagram. Both use blob tracking in TouchDesigner as a visual layer over their music. What grabbed me about their work was how physical it felt; your hands are literally revealing something on screen. But looking closer, the audio in their setups doesn't actually respond to the gestures. The music is just playing underneath. That gap felt like an obvious thing to close: same gesture input, driving both the visuals and the sound at the same time.

For System B, the reference was @jasminesokko's performance that went viral. She uses MediaPipe and Ableton to shape her voice in real time through hand gestures. The idea of treating your body as a controller for your own voice, turning a vocal performance into something more like instrument-playing, was what we wanted to explore and build from scratch.

Project 3: Product

---

System A: Gesture-Driven Audiovisual Filter Experience

System A leaned into Bre's strengths visually, so I told her to build the VFX however she wanted and that I'd figure out the audio around whatever gestures she used. Working reactively like that was actually more interesting than planning everything upfront together.

Bre built the VFX system in TouchDesigner using MediaPipe and a blob tracking workaround. Filters hidden behind a blur layer get revealed through a clarity filter based on where your hand is on screen. I took the same gesture data she was already reading; hand position and pinch, and mapped it to Ableton. Left hand X and Y go to Granulator III's sample position and stereo panning, changing where in the sample the sound plays and where it sits in the stereo field. Right hand pinch goes to stereo width through the Utility device and reverb amount through Echo simultaneously. Open hand, big and wet. Closed hand, mono and dry.

The thing that made System A actually work was that both the visuals and the audio were reading from the same input. Not layered separately; genuinely reacting to the same movement.

System B: Gesture-Controlled Vocal Performance System

System B was built around using your hands to shape your voice in real time. I handled the Ableton audio system and the TouchDesigner gesture routing. Bre built the visual UI layer that sits on top and gives the performer something to look at while performing.

The Ableton chain is: mic into Vocoder, with a MIDI keyboard providing the carrier tone. This creates a talking synth effect, where the synthesizer takes on the character of your voice. An open palm turns the vocoder on; a closed fist turns it off. After the Vocoder, the signal goes through Auto Pan set to Tremolo mode, where right hand movement controls the vibrato amount.

The performer I was referencing used plugins I didn't have access to, including a third-party tremolo controller. I also deliberately avoided external plugins so Bre could open and test the Ableton file on her own computer without running into compatibility issues. So I recreated everything with native Ableton devices. Auto Pan in Tremolo mode is a pretty decent substitute once you dial in the rate and shape settings.

The TD side follows the same basic architecture as System A: MediaPipe data through select and clamp nodes into the tdAbleton Package. The difference was figuring out which gestures to use. Originally I had hand tilt controlling the vibrato amount, but when we tested it we found that tilting your hand kept accidentally triggering the vocoder toggle at the same time. The gesture data overlapped more than I expected. I switched to tracking hand movement for vibrato instead, which kept the two channels clean.

Bre's UI gives the performer a visual on/off indicator anchored to the palm and a radial pattern that expands with tremolo intensity. It's not just decoration; it tells you what state the system is in while you're performing.

style="width: 90%; border: 5px solid grey; margin-top: 80px" class="w3-round" />